Head-up display

ABSTRACT

A head-up display for a vehicle. The head-up display comprises a projector and processor. The projector is arranged to project image content such that it is visible from an eye-box. The processor is arranged to receive captured images of a scene visible from the eye-box. The processor is arranged, at a first time to: detect a first object in a scene and instruct the image projector to project an icon (e.g. computer graphic) that appears, from the viewing position, to be coincident with the first object. The processor is further arranged to, at a second time later than the first time to: detect a second object in a line of sight from the eye-box position to the first object and instruct the image projector to change the visual appearance of the projected icon in response to the detection of the second object.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is claims priority to United Kingdom PatentApplication No. GB2202423.6 filed Feb. 22, 2022, which is herewithincorporated by reference into the present application.

FIELD

The present disclosure relates to a projector and a head-up display.More specifically, the present disclosure relates to a holographicprojector and a head-up display for a vehicle such as an automotivevehicle. The present disclosure also relates to a method of holographicprojection, a method of projecting a virtual image in a head-up displayand a method of displaying a virtual image on a window such as awindscreen using a head-up display.

BACKGROUND

Light scattered from an object contains both amplitude and phaseinformation. This amplitude and phase information can be captured on,for example, a photosensitive plate by well-known interferencetechniques to form a holographic recording, or “hologram”, comprisinginterference fringes. The hologram may be reconstructed by illuminationwith suitable light to form a two-dimensional or three-dimensionalholographic reconstruction, or replay image, representative of theoriginal object.

Computer-generated holography may numerically simulate the interferenceprocess. A computer-generated hologram, “CGH”, may be calculated by atechnique based on a mathematical transformation such as a Fresnel orFourier transform. These types of holograms may be referred to asFresnel or Fourier holograms. A Fourier hologram may be considered aFourier domain representation of the object or a frequency domainrepresentation of the object. A CGH may also be calculated by coherentray tracing or a point cloud technique, for example.

A CGH may be encoded on a spatial light modulator, “SLM”, arranged tomodulate the amplitude and/or phase of incident light. Light modulationmay be achieved using electrically-addressable liquid crystals,optically-addressable liquid crystals or micro-mirrors, for example.

The SLM may comprise a plurality of individually-addressable pixelswhich may also be referred to as cells or elements. The light modulationscheme may be binary, multilevel or continuous. Alternatively, thedevice may be continuous (i.e. is not comprised of pixels) and lightmodulation may therefore be continuous across the device. The SLM may bereflective meaning that modulated light is output from the SLM inreflection. The SLM may equally be transmissive meaning that modulatedlight is output from the SLM is transmission.

A holographic projector for imaging may be provided using the describedtechnology. Such projectors have found application in head-up displays,“HUD”, and head-mounted displays, “HMD”, including near-eye devices, forexample. Conventionally, a rectangular area (referred to herein as avirtual image area) is defined in the driver's field of view and thehead-up display may display image content in this rectangular area.

SUMMARY

Aspects of the present disclosure are defined in the appendedindependent claims.

Broadly, the present disclosure relates to image projection. It relatesto a method of image projection and an image projector which comprises adisplay device. The present disclosure also relates to a projectionsystem comprising the image projector and a viewing system, in which theimage projector projects or relays light from the display device to theviewing system. The present disclosure is equally applicable to amonocular and binocular viewing system. The viewing system may comprisea viewer's eye or eyes. The viewing system comprises an optical elementhaving optical power (e.g., lens/es of the human eye) and a viewingplane (e.g., retina of the human eye/s). The projector may be referredto as a ‘light engine’. The display device and the image formed (orperceived) using the display device are spatially separated from oneanother. The image is formed, or perceived by a viewer, on a displayplane. In some embodiments, the image is a virtual image and the displayplane may be referred to as a virtual image plane. In other embodiments,the image is a real image formed by holographic reconstruction and theimage is projected or relayed to the viewing plane. The image is formedby illuminating a diffractive pattern (e.g., hologram) displayed on thedisplay device.

The display device comprises pixels. The pixels of the display maydisplay a diffractive pattern or structure that diffracts light. Thediffracted light may form an image at a plane spatially separated fromthe display device. In accordance with well-understood optics, themagnitude of the maximum diffraction angle is determined by the size ofthe pixels and other factors such as the wavelength of the light.

There is also disclosed herein an improved HUD for an automotivevehicle. The HUD includes a picture generating unit. The picturegenerating unit may be arranged to generate a picture includinginformation content, such as speed or navigation information. There isalso provided an optical system arranged to form a virtual image of theinformation content. The virtual image of the information content may beformed at a suitable viewing position for the driver such as within thedriver's normal field of view whilst operating the automotive vehicle.For example, the virtual image of the information content may appear ata distance down the bonnet (or hood) of the vehicle from the driver. Thevirtual image of the information content is positioned so as not toadversely affect the driver's normal view of the scene. The virtualimage of the information content may be overlaid on the driver's view ofthe real world. The information content is computer-generated and may becontrolled or updated in real-time to provide real-time information tothe driver.

Embodiments relate to a picture generating unit comprises a holographicprojector by way of example only. The present disclosure is compatiblewith any display technology including a backlit liquid crystal display,a laser scanning display, a digital micro-mirror device “DMD”, afluorescent display and a plasma display. In embodiments relating to aholographic projector, the picture is a holographic reconstruction of acomputer-generated hologram. A HUD based on the holographic projectordescribed in full below is able to deliver a much greater contrast ratiothan currently available competing technologies because of theefficiency of the holographic process and its inherent suitability foruse with a laser light source.

The head-up display may comprise a holographic processor. The picturemay be a holographic reconstruction. The holographic processor may bearranged to output the computer-generated hologram to a spatial lightmodulator. The computer-generated hologram may be arranged to, at leastpartially, compensate for the shape of the windscreen of the vehicle.

The system may be arranged to form the virtual image of the pictureusing the windscreen by reflecting spatially-modulated light off thewindscreen. The light source may be a laser and/or the light of thepicture may be laser light. The spatial light modulator may be a liquidcrystal on silicon spatial light modulator. The picture may be formed byan interference process of the spatially-modulated light at the lightreceiving surface. Each computer-generated hologram may be amathematical transformation of a picture, optionally, a Fourier orFresnel transformation. The computer-generated hologram may be a Fourieror Fresnel hologram. The computer-generated hologram may be a hologramcomputer-generated by a point cloud method. The spatial light modulatormay be arranged to spatially-modulate the phase of the light from thelight source. The spatial light modulator may be arranged tospatially-modulate the amplitude of the light from the light source.

There is provided a head-up display for a vehicle having a window. Thehead-up display comprises a display device and an optical system. Insome embodiments, the display device is arranged to display a hologramof an image or picture for projection. The image or picture may be saidto comprise image or picture content. The image or picture content maycomprise a plurality of discrete computer graphics.

In a first group of embodiments, a holographic reconstruction of theimage is formed on a screen such as a diffuser by illuminating thedisplay device with light from a light source such as a laser diode. Inthese embodiments, the laser diode, display device and screen form apicture generating unit that will be familiar to the person skilled inthe art of holographic projection. In these embodiments, the opticalsystem may comprise an optical relay system, having at least one elementwith optical power, arranged to magnify the picture on the screen andproject it towards a windscreen of the vehicle to form a enlargedvirtual image of the picture. Such as a configuration has been disclosedin WO2020/016214, for example, which is incorporated herein in full byreference.

In a second group of embodiments, an intermediate reconstruction of thepicture is not formed on a screen and, instead, the hologram (morespecifically, light encoded with the hologram or spatially modulated inaccordance with the displayed hologram) is projected directly to theviewer. In these embodiments, it is sometimes said that the lens of theviewer's eye performs the hologram-to-image transformation—which may bea Fourier or Fresnel transformation, for example. In these embodiments,a pupil expander (or pair of orthogonal pupil expanders) may be employedto expand the eye-box. Such a configuration has been disclosed inGB2101666.2 filed 5 Feb. 2021, for example, which is incorporated hereinin full by reference.

According to a first aspect of the present disclosure, there is ahead-up display for a vehicle. The head-up display comprises a projectorand processor. The projector is arranged to project image content (suchthat it is) visible from an eye-box. The processor is arranged toreceive captured images of a scene visible from the eye-box. Theprocessor is arranged, at a first time to: detect a first object in ascene and instruct/drive the image projector to project an icon (e.g.computer graphic) that appears, from a viewing position (within theeye-box—i.e. an eye-box position), to be aligned/coincident with thefirst object. The processor is further arranged to, at a second timelater than the first time to: detect a second object in a line of sightfrom the viewing position to the first object and instruct/drive theimage projector to change the visual appearance of the projected icon inresponse to the detection of the second object.

The step of changing the visual appearance of the projected icon maycomprise changing at least one aspect of the physical form of theprojected icon.

The at least one physical form of the projected icon may be selectedfrom the group comprising: shape, colour, size and luminance.

The image projector may be a holographic projector comprising a spatiallight modulator arranged to display a hologram of the projected imagecontent. The holograms may be calculated in real-time.

The image content may be projected using an optical combiner such thatthe image content complements/adds to/overlays the scene visible fromthe eye-box.

The processor may be arranged to continually receive a viewing position(e.g. eye-box position) of a viewer within the eye-box and determine ifthe second object is in the line of sight based on a received viewingposition (e.g. eye-box position).

The first object may be a moving object. The processor may be arrangedto determine whether the second object is in the line of sight to thefirst object based on the position of the first object at the secondtime and, optionally, the viewing position (e.g. eye-box position) ofthe viewer at the second time.

The processor may be arranged to maintain positional/visual alignmentbetween the projected icon and first object, optionally, based on thereceived viewing position (e.g. eye-box position).

According to a second aspect of the present disclosure, there is adriver assistance system comprising a head-up display, a camera and auser-tracking system. The camera is arranged to capture images of ascene and continually output the captured images to the head-up display.The user-tracking system is arranged to monitor the position of the userof the head-up display and continually output the eye-box position ofthe user to the head-up display.

According to a third aspect of the present disclosure, there is a methodof head-up display. The method comprises a first step of capturing afirst image of a scene at a first time. The method comprises a secondstep of detecting a first object in the first image of the scene. Themethod comprises a third step of projecting an icon that appears, froman eye-box position, to be aligned/coincident with the first object. Themethod comprises a fourth step of capturing a second image of the sceneat a second time. The method comprises a fifth step of detecting asecond object in a line of sight from the user (e.g. eye-box position)to the first object. The method comprises a sixth step of changing thevisual appearance of the projected icon in response to the detection ofthe second object.

The method may further comprise changing the visual appearance of theprojected icon comprises changing at least one aspect of the physicalform of the projected icon. The at least one physical form of theprojected icon may be selected from the group comprising: shape, colour,size and luminance.

The term “hologram” is used to refer to the recording which containsamplitude information or phase information, or some combination thereof,about the object. The term “holographic reconstruction” is used to referto the optical reconstruction of the object which is formed byilluminating the hologram. The term “replay plane” is used herein torefer to the plane in space where the holographic reconstruction isfully formed. The term “replay field” is used herein to refer to thesub-area of the replay plane which can receive spatially-modulated lightfrom the spatial light modulator. The terms “image”, “replay image” and“image region” refer to areas of the replay field illuminated by lightforming the holographic reconstruction. In embodiments, the “image” maycomprise discrete spots which may be referred to as “image pixels”.

The terms “encoding”, “writing” or “addressing” are used to describe theprocess of providing the plurality of pixels of the SLM with a respectplurality of control values which respectively determine the modulationlevel of each pixel. It may be said that the pixels of the SLM areconfigured to “display” a light modulation distribution in response toreceiving the plurality of control values. Thus, the SLM may be said to“display” a hologram.

It has been found that a holographic reconstruction of acceptablequality can be formed from a “hologram” containing only phaseinformation related to the original object. Such a holographic recordingmay be referred to as a phase-only hologram. Embodiments relate to aphase-only hologram but the present disclosure is equally applicable toamplitude-only holography.

The present disclosure is also equally applicable to forming aholographic reconstruction using amplitude and phase information relatedto the original object. In some embodiments, this is achieved by complexmodulation using a so-called fully complex hologram which contains bothamplitude and phase information related to the original object. Such ahologram may be referred to as a fully-complex hologram because thevalue (grey level) assigned to each pixel of the hologram has anamplitude and phase component. The value (grey level) assigned to eachpixel may be represented as a complex number having both amplitude andphase components. In some embodiments, a fully-complexcomputer-generated hologram is calculated.

Reference may be made to the phase value, phase component, phaseinformation or, simply, phase of pixels of the computer-generatedhologram or the spatial light modulator as shorthand for “phase-delay”.That is, any phase value described is, in fact, a number (e.g. in therange 0 to 2π) which represents the amount of phase retardation providedby that pixel. For example, a pixel of the spatial light modulatordescribed as having a phase value of π/2 will change the phase ofreceived light by π/2 radians. In some embodiments, each pixel of thespatial light modulator is operable in one of a plurality of possiblemodulation values (e.g. phase delay values). The term “grey level” maybe used to refer to the plurality of available modulation levels. Forexample, the term “grey level” may be used for convenience to refer tothe plurality of available phase levels in a phase-only modulator eventhough different phase levels do not provide different shades of grey.The term “grey level” may also be used for convenience to refer to theplurality of available complex modulation levels in a complex modulator.

Although different embodiments and groups of embodiments may bedisclosed separately in the detailed description which follows, anyfeature of any embodiment or group of embodiments may be combined withany other feature or combination of features of any embodiment or groupof embodiments. That is, all possible combinations and permutations offeatures disclosed in the present disclosure are envisaged.

Although reference is made to a head-up display for a vehicle, theskilled person will understand that the present disclosure extends tohead-up display for other purposes and the device may more generally bereferred to as a display system.

In the present disclosure, the term “substantially” when applied to astructural units of an apparatus may be interpreted as the technicalfeature of the structural units being produced within the technicaltolerance of the method used to manufacture it.

BRIEF DESCRIPTION OF THE FIGURES

Specific embodiments are described by way of example only with referenceto the following figures:

Specific embodiments are described by way of example only with referenceto the following figures:

FIG. 1 is a schematic showing a reflective SLM producing a holographicreconstruction on a screen;

FIG. 2 illustrates a block diagram of a display system in accordancewith some embodiments;

FIG. 3 illustrates a block diagram of an AR application in accordancewith some embodiments; and

FIGS. 4A, 4B, 4C 4D and 4E illustrate an example AR environment in whichthe physical form of virtual content associated with a detected physicalobject is changed in response to detection of an impeded line of sight.

The same reference numbers will be used throughout the drawings to referto the same or like parts.

DETAILED DESCRIPTION

The present invention is not restricted to the embodiments described inthe following but extends to the full scope of the appended claims. Thatis, the present invention may be embodied in different forms and shouldnot be construed as limited to the described embodiments, which are setout for the purpose of illustration.

Terms of a singular form may include plural forms unless specifiedotherwise.

A structure described as being formed at an upper portion/lower portionof another structure or on/under the other structure should be construedas including a case where the structures contact each other and,moreover, a case where a third structure is disposed there between.

In describing a time relationship—for example, when the temporal orderof events is described as “after”, “subsequent”, “next”, “before” orsuchlike—the present disclosure should be taken to include continuousand non-continuous events unless otherwise specified. For example, thedescription should be taken to include a case which is not continuousunless wording such as “just”, “immediate” or “direct” is used.

Although the terms “first”, “second”, etc. may be used herein todescribe various elements, these elements are not to be limited by theseterms. These terms are only used to distinguish one element fromanother. For example, a first element could be termed a second element,and, similarly, a second element could be termed a first element,without departing from the scope of the appended claims.

Features of different embodiments may be partially or overall coupled toor combined with each other, and may be variously inter-operated witheach other. Some embodiments may be carried out independently from eachother, or may be carried out together in co-dependent relationship.

Optical Configuration

FIG. 1 shows an embodiment in which a computer-generated hologram isencoded on a single spatial light modulator. The computer-generatedhologram is a Fourier transform of the object for reconstruction. It maytherefore be said that the hologram is a Fourier domain or frequencydomain or spectral domain representation of the object. In thisembodiment, the spatial light modulator is a reflective liquid crystalon silicon, “LCOS”, device. The hologram is encoded on the spatial lightmodulator and a holographic reconstruction is formed at a replay field,for example, a light receiving surface such as a screen or diffuser.

A light source 110, for example a laser or laser diode, is disposed toilluminate the SLM 140 via a collimating lens 111. The collimating lenscauses a generally planar wavefront of light to be incident on the SLM.In FIG. 1 , the direction of the wavefront is off-normal (e.g. two orthree degrees away from being truly orthogonal to the plane of thetransparent layer). However, in other embodiments, the generally planarwavefront is provided at normal incidence and a beam splitterarrangement is used to separate the input and output optical paths. Inthe embodiment shown in FIG. 1 , the arrangement is such that light fromthe light source is reflected off a mirrored rear surface of the SLM andinteracts with a light-modulating layer to form an exit wavefront 112.The exit wavefront 112 is applied to optics including a Fouriertransform lens 120, having its focus at a screen 125. More specifically,the Fourier transform lens 120 receives a beam of modulated light fromthe SLM 140 and performs a frequency-space transformation to produce aholographic reconstruction at the screen 125.

Notably, in this type of holography, each pixel of the hologramcontributes to the whole reconstruction. There is not a one-to-onecorrelation between specific points (or image pixels) on the replayfield and specific light-modulating elements (or hologram pixels). Inother words, modulated light exiting the light-modulating layer isdistributed across the replay field.

In these embodiments, the position of the holographic reconstruction inspace is determined by the dioptric (focusing) power of the Fouriertransform lens. In the embodiment shown in FIG. 1 , the Fouriertransform lens is a physical lens. That is, the Fourier transform lensis an optical Fourier transform lens and the Fourier transform isperformed optically. Any lens can act as a Fourier transform lens butthe performance of the lens will limit the accuracy of the Fouriertransform it performs. The skilled person understands how to use a lensto perform an optical Fourier transform.

Hologram Calculation

In some embodiments, the computer-generated hologram is a Fouriertransform hologram, or simply a Fourier hologram or Fourier-basedhologram, in which an image is reconstructed in the far field byutilising the Fourier transforming properties of a positive lens. TheFourier hologram is calculated by Fourier transforming the desired lightfield in the replay plane back to the lens plane. Computer-generatedFourier holograms may be calculated using Fourier transforms.Embodiments relate to Fourier holography and Gerchberg-Saxton typealgorithms by way of example only. The present disclosure is equallyapplicable to Fresnel holography and Fresnel holograms which may becalculated by a similar method. In some embodiments, the hologram is aphase or phase-only hologram. However, the present disclosure is alsoapplicable to holograms calculated by other techniques such as thosebased on point cloud methods. United Kingdom application No. GB2112213.0 filed 26 Aug. 2021, incorporated herein by reference,discloses example hologram calculation methods that may be combined withthe present disclosure.

In some embodiments, there is provided a real-time engine arranged toreceive image data and calculate holograms in real-time using thealgorithm. In some embodiments, the image data is a video comprising asequence of image frames. In other embodiments, the holograms arepre-calculated, stored in computer memory and recalled as needed fordisplay on a SLM. That is, in some embodiments, there is provided arepository of predetermined holograms.

Light Modulation

The display system comprises a display device defining the exit pupil ofthe display system. The display device is a spatial light modulator. Thespatial light modulation may be a phase modulator. The display devicemay be a liquid crystal on silicon, “LCOS”, spatial light modulator.

AR-HUD

Augmented Reality, “AR”, systems may be utilized in a multiplicity ofinstances. One exemplary use for AR is to aid users while operating avehicle. For instance, virtual content may be presented on a HUD toprovide a user with directions to a desired destination. Virtual arrowsor other indicators may be presented on the HUD to augment the user'sphysical world and provide a route the user should follow to reach theirdesired destination. As another example, informational text may bepresented on the HUD that describes nearby stores, vehicles, etc. Whileit is contemplated that AR provides valuable information, presentinginformation on a HUD presents challenges due to the continuouslychanging environment. The distances between the vehicle and surroundingobjects change as the vehicle and/or the surrounding objects move.

AR allows a user to augment reality with virtual content. Virtualcontent may be presented on a transparent display of a viewing device toaugment the user's real-world environment. As an example, virtualcontent presented on a HUD in an automobile can present the user witharrows, shapes, 3D objects, other indicators, and or other illustrationsthat may provide the user with directions to a desired destination,and/or other information with respect to the environment. As anotherexample, virtual content describing vehicles and/or businesses can bepresented on the HUD to provide a user with additional informationregarding their environment.

To augment the reality of a user, virtual content may be presented onthe HUD to create the appearance that the virtual content is present inthe user's real-world environment rather than just presented arbitrarilyon a display. To properly create this appearance, a viewing deviceadjusts a rendering of the virtual content corresponding to a physicalobject.

A display system is used to augment the reality of a user. The user maybe a human user (e.g., a human being), a machine user (e.g., a computerconfigured by a software program to interact with the viewing device),or any suitable combination thereof (e.g., a human assisted by a machineor a machine supervised by a human). The display system is a computingdevice integrated in a vehicle, such as an automobile, to providevirtual content on a head-up display (HUD).

The display system may comprise a transparent or semi-transparent screenwhich may be the windshield of a car housing the display system or anoptical combiner, such as pop-up combiner, of a stand-alone head-updisplay. The user may simultaneously view virtual content presented bythe display system as well as a physical objects in the user's field ofview of the real-world physical environment.

The display system may provide the user with an augmented realityexperience. For example, the display system can present virtual contentthat the user can view in addition to physical objects that are in thefield of view of the user in the real-world physical environment.Virtual content can be any type of image, animation, etc., presented onthe display. For example, virtual content can include a virtual model(e.g., 3D model) of an object or a simple indicia such as a warningtriangle and similar shape.

The physical object may include any type of identifiable objects such asa 2D physical object (e.g., a picture), a 3D physical object (e.g., avehicle, cyclist, pedestrian, building, street, etc.), a location (e.g.,at the bottom floor of a factory), or any references (e.g., perceivedcorners of walls or furniture) in the real-world physical environment.

The display system can present virtual content in response to detectingone or more identified objects (e.g., physical object) in the physicalenvironment. For example, the display system may include optical sensorsto capture images of the real-world physical environment and computervision recognition to identify physical objects.

In one example embodiment, the display system locally analyses capturedimages using a local content dataset or any other dataset previouslystored by the display system. The local content dataset may include alibrary of virtual content associated with real-world physical objectsor references. For example, the local content dataset can include imagedata depicting real-world physical objects. The display system canutilize the captured image of a physical object to search the localcontent dataset to identify the physical object and its correspondingvirtual content.

In one example, the display system can analyse an image of a physicalobject to identify feature points of the physical object. The displaysystem can utilize the identified feature points to identify acorresponding real-world physical object from the local content dataset.The display system may also identify tracking data related to thephysical object (e.g., GPS location of the viewing device, orientation,distance to the physical object).

If the captured image is not recognized locally, the display system candownload additional information (e.g., virtual content) corresponding tothe captured image, from a database of a server over a network, forexample.

In another example, a physical object in the image is tracked andrecognized remotely at the server using a remote dataset or any otherpreviously stored dataset of the server. The remote content dataset mayinclude a library of virtual content or augmented information associatedwith real-world physical objects or references. In this type ofembodiment, the display system can provide the server with the capturedimage of the physical object. The server can use the received image toidentify the physical object and its corresponding virtual content. Theserver can then return the virtual content to the viewing device.

The display system can project the virtual content to augment thereality of the user. For example, the display system can present thevirtual content to allow the user to simultaneously view the virtualcontent as well as the real-world physical environment in the field ofview.

As an example, the display system can change a visual property of thevirtual content (e.g. shape) corresponding to a cyclist as anothervehicle crosses the user's field of view or line of sight to thecyclist. As another example, the display system can change the colour orsize of virtual content in response to the same scenario.

The display system can present the virtual content at a position thatcorresponds to the location of the physical object as perceived by auser. Accordingly, the virtual content appears to the user to be nearbyor overlapping the physical object.

The display system continuously updates the presentation of the virtualcontent based on the location of the physical object in relation to theuser by re-rendering the virtual content based on changes of thelocation. As a result, the user may perceive the virtual content to befixed in a location of the user's real-world environment as the usermoves.

Any of the machines, databases, or devices disclosed herein may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform oneor more of the functions described herein for that machine, database, ordevice. As used herein, a “database” is a data storage resource and maystore data structured as a text file, a table, a spreadsheet, arelational database (e.g., an object-relational database), a triplestore, a hierarchical data store, or any suitable combination thereof.Moreover, any two or more of the machines, databases, or devices may becombined into a single machine, and the functions described herein forany single machine, database, or device may be subdivided among multiplemachines, databases, or devices.

The network may be any network that enables communication between oramong machines (e.g., server), databases, and devices (e.g., head-updisplays). Accordingly, the network may be a wired network, a wirelessnetwork (e.g., a mobile or cellular network), or any suitablecombination thereof. The network may include one or more portions thatconstitute a private network, a public network (e.g., the Internet), orany suitable combination thereof.

FIG. 2 illustrates a block diagram in accordance with embodiments. Thedisplay system 102 includes sensors 202, a transparent display 204, acomputer processor 208, and a storage device 206. The display system 102is integrated into a vehicle, such as an automobile, motorcycle, plane,boat, recreational vehicle (RV), etc.

The sensors 202 can include any type of known sensors. The sensors 202include at least one infrared or visible light image capture device(e.g. camera) arranged to capture images of the scene at, for example,video rate.

The transparent display 204 includes, for example, a display configuredto display holograms of virtual images generated and calculated by theprocessor 208. The transparent display 204 can be positioned such thatthe user can simultaneously view virtual content presented on thetransparent display and a physical object in a field of view of theuser. For example, the transparent display 204 can be a HUD in anautomobile or other vehicle that presents virtual content on awindshield of the vehicle while also allowing a user to view physicalobjects through the windshield. For example, the HUD can be configuredto display virtual images itself or, alternatively, can presentedvirtual images projected onto the HUD.

The processor 208 includes an AR application 210 configured to presentvirtual content on the transparent display 204 to augment the reality ofthe user. The AR application 210 can receive data from sensors 202(e.g., an image of the physical object, location data, etc.), and usethe received data to identify at least one physical object (e.g.cyclist) and project virtual content (e.g. a warning shape) using thetransparent display 204.

To identify the physical object (e.g. cyclist), the AR application 210determines whether an image captured by the display system 102 matchesan image locally stored by the display system 102 in the storage device206. The storage device 206 can include a local content dataset ofimages and corresponding virtual content. For example, the displaysystem 102 can receive a content data set from the server 110, and storethe received content data set in the storage device 206.

The AR application 210 can compare a captured image of the physicalobject to the images locally stored in the storage device 206 toidentify the physical object. For example, the AR application 210 cananalyse the captured image of a physical object to identify featurepoints of the physical object. The AR application 210 can utilize theidentified feature points to identify the physical object from the localcontent dataset. In some embodiments, the AR application 210 canidentify a physical object based on characterising features of theobject.

If the AR application 210 cannot identify a matching image from thelocal content dataset, the AR application 210 may provide the capturedimage of the physical object to the server 110. The server 110 uses thecaptured image to search a remote content dataset maintained by theserver 110.

The remote content dataset maintained by the server can be larger thanthe local content dataset maintained by the display system 102. Forexample, the local content dataset maintained by the display system 102can include a subset of the data included in the remote content dataset,such as a core set of images or the most popular images determined bythe server.

Once the physical object (e.g. cyclist) has been identified by eitherthe display system 102 or the server, the corresponding virtual contentcan be retrieved and projected on the transparent display 204 to augmentthe reality of the user by displaying the virtual content so that thevirtual content is overlain on the real-world view of the user throughthe transparent display. The AR application 210 can present the virtualcontent on the transparent display 204 to, for example, highlight thephysical object (e.g. cyclist) to the user—i.e. draw the user'sattention to the cyclist. For example, the AR application 210 canpresent a shape or other indicator that are overlain with the physicalobject (e.g. cyclist).

Virtual Content Change in Response to a Hidden Object

As described in the following, the AR application 210 adjusts one ormore properties or parameters of the virtual content based on detectionof another object between the physical object (e.g. cyclist) and theviewer. Adjusting the properties or parameters results in the virtualcontent being displayed with a different property (e.g. shape or colour)when an intervening object is detected. That is, the virtual content hasa changed or modified appearance.

In an embodiment, the AR application 210 changes the shape of thevirtual content corresponding to a cyclist as a car blocks the user'sview of the cyclist—e.g. blocks the field of view such as crosses theline of sight. Accordingly, the physical form of the virtual contentpresented on the transparent display 204 becomes different when the carinterferes with the users 106 view of the cyclist. As another example,the AR application 210 can change a colour of virtual contentcorresponding to the cyclist as the car moves into the line of sight.The virtual content may have a first form when the user has an unimpededview of the physical object (e.g. cyclist) and the virtual content mayhave a second form when the user's view of the physical object (e.g.cyclist) is impeded.

The AR application 210 may continuously updates presentation of thevirtual content based on the location of the physical object (e.g.cyclist) in relation to the other vehicle and/or the user. As the othervehicle and physical object move with respect to each other, new datamay be used by the AR application 210 to re-render the virtual contenton the transparent display 204, at display positions that correspond tothe new location data.

The AR application 210 may update presentation of the virtual content asthe vehicle and/or physical object change positions. For example, the ARapplication 210 can gather updated sensor data from the sensors 202 asthe vehicle moves and determine an updated position of the physicalobject in relation to the vehicle. The AR application 210 updatespresentation of the virtual content based on the determined updatedposition of the physical object in relation to the vehicle. For example,the AR application 210 adjusts a display shape of the virtual contentbased on the updated position of the physical object. The AR application210 presents the updated presentation of the virtual content on thetransparent display 204, thereby providing the user with a changeddepiction of the virtual content.

Any one or more of the modules described herein may be implemented usinghardware (e.g., a processor of a machine) or a combination of hardwareand software. For example, any module described herein may configure aprocessor to perform the operations described herein for that module.Moreover, any two or more of these modules may be combined into a singlemodule, and the functions described herein for a single module may besubdivided among multiple modules. Furthermore, according to variousexample embodiments, modules described herein as being implementedwithin a single machine, database, or device may be distributed acrossmultiple machines, databases, or devices.

FIG. 3 illustrates a block diagram of an example embodiment of an ARapplication 210, according to some embodiments. To avoid obscuring theinventive subject matter with unnecessary detail, various functionalcomponents (e.g., modules) that are not germane to conveying anunderstanding of the inventive subject matter have been omitted fromFIG. 3 . However, a skilled artisan will readily recognize that variousadditional functional components may be supported by the AR application210 to facilitate additional functionality that is not specificallydescribed herein. Furthermore, the various functional modules depictedin FIG. 3 may reside on a single computing device or may be distributedacross several computing devices in various arrangements such as thoseused in cloud-based architectures.

As shown, the AR application 210 includes an input module 302, anidentification module 304, a position determination module 306, aline-of-sight determination module 308, a content generation module 310and a display module 312.

The input module 302 receives sensor data from sensors 202, sensor datamay include, for example, and without limitation optical image data ofthe physical object, ToF data, imaged light patterns,location/positional data, other data associated with an operation of thevarious sensors, and a combination thereof. The input module 302provides the received sensor data to any of the other modules includedin the AR application 210.

The identification module 304 identifies a physical object andcorresponding virtual content based on an image of the physical objectcaptured by sensors 202 of the display system. For example, theidentification module 304 can determine whether the captured imagematches or is similar to an image locally stored by the display systemin the storage device 206.

The identification module 304 compares a captured image of the physicalobject to a local content dataset of images locally stored in thestorage device 206 to identify the physical object. For example, theidentification module 304 can analyse the captured image of a physicalobject to identify feature points of the physical object. Theidentification module 304 can utilize the identified feature points toidentify the physical object from the local content dataset.

If the identification module 304 cannot identify a matching image fromthe local content dataset, the identification module 304 can provide thecaptured image of the physical object to the server and the server cansearch a remote content dataset maintained by the server.

Once the physical object 104 has been identified, the identificationmodule 304 can access the corresponding virtual content to be presentedon the transparent display 204 to augment the reality of the user.

The position determination module 306, determines the position of thephysical object in relation to the display system. The positiondetermination module 306 can analyse images of the physical object todetermine the position of the physical object in relation to the displaysystem. For example, the position determination module 306 can analyseimages captured by the sensors 202 and identify the physical object inthe captured image. The position determination module 306 thendetermines the position of the physical object in relation to thedisplay system based on the location of the physical object in thecaptured image.

The line-of-sight module 308 uses the position determined by theposition determination module 306 and, for example, eye-trackinginformation of the user to determine information related to aline-of-sight from the user to the physical object (e.g. cyclist). Theline-of-sight module 308 may use any suitable technique to identify aline-of-sight and detect if another object (e.g. a car) is obscuring(e.g. impeding, partially blocking or fully blocking) the user's view ofthe identified physical object (e.g. cyclist). By way of example only,the line-of-sight module 308 may determine that the line-of-sight to thephysical object is blocked if the physical object is no longerdetectable by the identification module 304. The line-of-sight module308 may provide a first output if the line-of-sight is clear and asecond output if the line-of-sight is impeded. Reference herein to theeye-tracking or user-tracking is by way of example only of one method ofdetermining if the user's view is blocked and it is not essential thatthe present invention utilises eye-tracking or user-tracking informationto function as disclosed herein. In some embodiments, the step ofdetermining whether the line-of-sight is block is based on a fixedviewing position—e.g. fixed eye-box position such as the centre of theeye-box.

The content generation module 310 generates virtual content based on theoutput of the line-of-sight module 308. For example, the contentgeneration module 310 changes the display form (e.g. shape) of thevirtual content if the output of the line-of-sight module 308 changes.

The display module 312 renders the virtual content on the transparentdisplay 204. This can include virtual content intended to augmentphysical objects visible through the transparent display 204. In someembodiments, the display module 312 calculates a hologram of the outputof the content generation module 310. The display module 312 can renderthe virtual content based on the position of a physical objectcorresponding to the virtual content. For example, the display module312 can render the virtual content at a display position on thetransparent display 204 that causes the virtual content to appear asoverlapping and/or near the physical object to a user.

The display module 312 continuously updates rendering of virtual contenton the transparent display 204. For example, the display module 312updates the display of the virtual content as the depth and/or positionof a physical object 104 changes. Accordingly, the virtual contentappears to be a part of the user's real-world environment and pairedwith its corresponding physical object. In some embodiments, hologramsare calculated in real-time.

FIGS. 4A to 4E illustrate an embodiment of the present disclosure by wayof example only. FIGS. 4A to 4E show the upper boundary 401 and lowerboundary 403 of a field of view 405 defined by a vehicle windscreen.FIGS. 4A to 4E also show a cyclist 407 (which corresponds to thephysical object of the prior description) and another car 409 which ismoving towards the cyclist 407. FIG. 4A shows the events before thedisplay system detects the cyclist. FIG. 4B shows the virtual content411 (having a diamond shape in this example) projected by the projectionsystem of the present disclosure. It is said that the virtual content411 corresponds to the cyclist 407. The virtual content 411 draws theuser's attention to the cyclist 407 at a road junction in this example.FIG. 4C shows the car 409 turning towards the cyclist 407 but notimpeding the user's view of the cyclist 407. The virtual content 411 istherefore still a diamond shape. As the car 409 manoeuvres further, itmoves to a position or positions that imped the user's view of thecyclist 407 (FIG. 4D). This is detected by the system of the presentdisclosure and, in response, the physical form of the virtual content411′ is changed. The changed virtual content 411′ may have the form of ablind spot icon which will be familiar to the person skilled in the artof driver assistance systems. As the car manoeuvres away from thecyclist, line-of-sight is restored and, optionally, the virtual content411 returns to its original form (FIG. 4E). It can therefore beunderstood that the virtual content has a first form 411 when thecorresponding line-of-sight is clear and a second form 411′ when theline-of-sight is not clear. These features provide the user with avisual warning that a physical object is hidden, out of sight, behindthe car 409. For the avoidance of doubt, in FIG. 4D, the cyclist is notvisible because it is behind the car (from the viewer's perspective) butthe virtual content 411′ is visible because it is overlain on thereal-world scene by the projector.

Additional Features

The methods and processes described herein may be embodied on acomputer-readable medium. The term “computer-readable medium” includes amedium arranged to store data temporarily or permanently such asrandom-access memory (RAM), read-only memory (ROM), buffer memory, flashmemory, and cache memory. The term “computer-readable medium” shall alsobe taken to include any medium, or combination of multiple media, thatis capable of storing instructions for execution by a machine such thatthe instructions, when executed by one or more processors, cause themachine to perform any one or more of the methodologies describedherein, in whole or in part.

The term “computer-readable medium” also encompasses cloud-based storagesystems. The term “computer-readable medium” includes, but is notlimited to, one or more tangible and non-transitory data repositories(e.g., data volumes) in the example form of a solid-state memory chip,an optical disc, a magnetic disc, or any suitable combination thereof.In some example embodiments, the instructions for execution may becommunicated by a carrier medium. Examples of such a carrier mediuminclude a transient medium (e.g., a propagating signal that communicatesinstructions).

It will be apparent to those skilled in the art that variousmodifications and variations can be made without departing from thescope of the appended claims. The present disclosure covers allmodifications and variations within the scope of the appended claims andtheir equivalents.

1. A head-up display for a vehicle, the head-up display comprising: aprojector arranged to project image content visible from an eye-box; aprocessor arranged to receive captured images of a scene visible fromthe eye-box and, at a first time: detect a first object in a scene anddrive the image projector to project an icon that appears, from aviewing position, to be substantially aligned with the first object,wherein the processor is further arranged to, at a second time laterthan the first time: detect a second object in a line of sight from theviewing position to the first object and drive the image projector tochange the visual appearance of the projected icon in response to thedetection of the second object.
 2. A head-up display as claimed in claim1 wherein changing the visual appearance of the projected icon compriseschanging at least one aspect of the physical form of the projected icon.3. A head-up display as claimed in claim 2 wherein the at least onephysical form of the projected icon is selected from the groupcomprising: shape, colour, size and luminance.
 4. A head-up display asclaimed in claim 1 wherein the image projector is a holographicprojector comprising a spatial light modulator arranged to display ahologram of the projected image content.
 5. A head-up display as claimedin claim 1 wherein the image content is projected using an opticalcombiner such that the image content complements/adds to/overlays thescene visible from the eye-box.
 6. A head-up display as claimed in claim1 wherein the processor is arranged to continually receive the viewingposition of a viewer and determine if the second object is in the lineof sight based on a received viewing position.
 7. A head-up display asclaimed in claim 6 wherein the first object is a moving object, and theprocessor is arranged to determine whether the second object is in theline of sight to the first object based on the position of the firstobject at the second time and, optionally, the viewing position of theviewer at the second time.
 8. A head-up display as claimed in claim 6wherein the processor is arranged to maintain positional alignmentbetween the projected icon and first object based on the receivedviewing position.
 9. A driver assistance system comprising: a head-updisplay of any preceding claim; a camera arranged to capture images of ascene and continually output the captured images to the head-up display;and a user-tracking system arranged to monitor the position of the userof the head-up display and continually output the viewing position ofthe viewer to the head-up display.
 10. A method of head-up displaycomprising: capturing a first image of a scene at a first time;detecting a first object in the first image of the scene; projecting anicon that appears, from a viewing position, to be substantially alignedwith the first object; capturing a second image of the scene at a secondtime; detecting a second object in a line of sight from the viewer tothe first object; and changing the visual appearance of the projectedicon in response to the detection of the second object.
 11. A method ofhead-up display as claimed in claim 10 wherein changing the visualappearance of the projected icon comprises changing at least one aspectof the physical form of the projected icon.
 12. A method of head-updisplay as claimed in claim 11 wherein the at least one physical form ofthe projected icon is selected from the group comprising: shape, colour,size and luminance.